These settings relate to the file system and storage.
File Upload Limits
These setting govern the size of files that can be uploaded to the application. They usually affect Model, since users upload large data files for processing. Buffering governs the size of the 'chunks' that the uploader will accept each time a piece of the source file is uploaded.
Note that setting the file size too high can overload the web servers and client browser. If large files need to be imported it is usually better to use a network reference to those files instead.
Permanent Storage
The file system is where content that is generated from Pyramid is stored PERSISTENTLY. Using these settings, admins can adjust the venue where permanent files, generated from the modeling and publication engines, are stored. This functionality is key in elastic deployments (like Kubernetes), where servers are added and removed consistently.
It should be noted that this does not describe the handling of temporary files or those used in transit. This are handled automatically and internally by the platform.
The file system can be configured with these options:
- Internal File System (default): all temporary and persistent content is stored on the designated data drives for each server as specified in the installation process.
- Share Folder: all persistent files are stored on the specified shared folder or directory
- AWS S3: all persistent files are stored on AWS S3 Bucket storage
- Azure Blob: all persistent files are stored on Azure Blob storage
- FTP: all persistent files are stored on FTP file system.
- NFS: all persistent files are stored on NFS file system. Click here to learn how to set up NFS.
In Kubernetes deployments, only AWS S3, Azure Blog, FTP, or NFS storage can (and must) be used.
Note: this feature is available with the Enterprise Edition license only.
Storage Configurations
Shared Folder
Use of a shared folder differs between Windows and Linux. In both scenarios, the shared folder needs to be IDENTICALLY deployed to each server in the cluster, pointing to a common shared resource.
Windows
- Folder: supply the UNC path to a shared folder
To create the shared folder in Windows, the UNC path needs to be the same on all servers. It can also be created with a net use command that adds a virtual drive to each Windows machine with the following command:
net use Z: \\server\folder /user:domain\myusername mypassword /persistent:yes
Linux
- Folder: supply the local path that is linked to a shared mounted drive
To create the shared folder in Linux, mount a folder from a server to a local folder on all servers in the cluster. This can be done with the following command:
sudo mount -t nfs servername:/sharedfolder /mnt/shared
AWS S3
- Region: select an AWS region from the list
- Authentication: provide the AWS access Key ID and secret access keys
If the details are correct, you should be able to pick a bucket from the drop down list.
Azure Blob
- Authentication: provide the Azure account and keys
If the details are correct, you should be able to pick a container from the drop down list.
Test
To check the storage works, click the test button before committing the changes.
Transitioning File Systems
If the file system is transitioned, old files are NOT migrated to the new destination. As such, its recommended that the file system is setup as soon as possible when deploying Pyramid.